DOCUHEIT BESOHE 



ED 127 929 



IR 003 891 



AOTHOn 
TITLE 

SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



ri/ones, Ie^> ; And Others 

A Plan for the Evaluation of a Project to Develop 
Basic Hedical sciences Lessons on PIATO IV. 
National Institutes of Health (DHEW) , Bethesda^ Hd. . 
Bureau of Health Manpower Education. 
Aug 76 

NIH-BHME-N01-PE-34068 

7p. ; Paper presented at the Association for the 
Development of Computer-Based Instructional Systems 
Summer Conference (Hinneapolis^ Hinnesota, August 
10-12^ 1976) 

MF-$0.83 HC-$1.67 Plus Postage. 

♦Computer Assisted Instruction; Ediscational Research; 
Evaluation; *Evaluation Methods; *Experimental 
Programs; Higher Education; Instructional Materials; 
Instructional Technology; *Hedical Education; 
♦Program Evaluation; Project Applications; 
Statistical Analysis 

ADCIS 76; PLATO IV; Programmed Logic for Automatic 
Teaching Operations 



A projec-. to introduce PLATO IV computer-assisted 
instruction (CAI) in medical sciences education for health 
professionals was implemented at the School of Basic Hedical sciences 
at the University of " Illinois. This paper describes the plan for 
evaluation of the project. Using a student questionnaire and 
additional general questions, the effectiveness of courseware 
materials will be assessed. Usage will be determined by the total 
hours spent with instructional materials and the number of sessions 
on the computer. Correlations will be calculated using test scores 
and hours of use on the appropriate lessons. If the number of total 
hours or total sessions separates students into two or more groups, 
analysis of variance will be conducted on the scores. The evaluation 
is intended to provide information for decision making and 
information for project personnel. (CH) 
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Introduction 

^ For three years a federally funded PLATO IV Computer-based Medical Education 
Project* has been operational. Briefly, the purpose of this Project Is to 
implement a PLATO IV computer-assisted instructional system that covers a major 
segment of basic medical sciences education for health professionals. The 
Project is adapted directly from the successful basic medical science program 
rsj at the School of Basic Medical Sciences at Urbana-Champaign (SBMS-UC) ^ College 
T-H of Medicine, University of Illinois. The "goal cf the Project Is to present the 
content in small moiules complete with educational objectives and assessment 
y procedures by which students can judg6 their grasp of the material. This 
approach allows self-paced leaml g and provides immediate feedback ^o the 
$i)tudent about performances within the nodule. The Project has been described 
previously (Hody, 1975; Hody et al, , 1975). this paper describes the plan for 
evaluating the Project. 

Purposes of the Evaluation — A General Statement 

One purpose of this evaluation is to examine the progress of the Computer-based 
Education Project toward goals articulated in documentatiou describing its incep- 
tion. This requires examining the Project's attainments and assessing the impact 
of the Project's activities. Another purpose is to provide organized evaluated 
feedback to individuates considering participating in this or a similar endeavor, 
to the co^i&minlty of medical educators, and to the agency supporting the Project's 
work. The main focus, however, is to provide to the funding agency Infoxrmation 
for decision-making. 

In addition, we hope to highlight for Project personnel those aspects of the 
Project for which attainments fall short, as well as those for which goals have 
been met. It is hoped that evaluative information from this report will facil^ 
itate planning future activities by indicating where progress is discrepant 
from goals and where objectives have been achieved. 

Theoretical Underpinnings 

The conduct of an evaluation is a complex task in that there are many purpose 
for providing evaluative feedback, many audiences — with varying nee^s— inter- 
ested in the data, many facets of the entity being evaluated aiid diverse limi- 
tations placed on the evaluators in terms of time, resources, axid support*. An 
evaluation and subsequent report cannot address all audiences, meet the needs 
of all Individuals, and focus on every aspect of £he entity being evaluated. 
Because of this, it is desirable to provide a fram\swork for the present evalu-* 
ation activities. This can be accompli.sbed by describing briefly the founda-- 
tions upon which this plan is built. In doing so, the perspectives of the 
evaluators will become clear and will provide reasonable expectations for what 
follows. 

to ■ ■ ■■■ v: 
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*The efforts of the Project and portions of Its evaluation are pursuant to 
Contract No. NOl-PE-34068 with National Inbtitutes of Health, Bureau of Health 
Manpower Education. 
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Information for Decision-Making . Self-paced learning, self -assessment, com- 
puterizcd learning modules, and Instructional management all have implications 
for changing, the role of Instruction In medical education. One of the poten- 
tial long-term contributions of the Project lies in providing an examination 
of these factors. For example, educators require information upon which to 
assess the contribution of the Project »s activities to medical education. 
Further, since the adaptation of a first yeur basic science curriculum to the 
PLATO IV system has required considerable financial support and demanded 
extensive human commitment, individuals cohtmplating joining the network of 
schools developing medical materials will ask questions about costs, require- 
ments in terms of personnel, and hardware and software needed to support 
developmental activities. Answers to these questions will then be weighed 
aghast pott;atlal contributions to arrive at a decision about partxclpating. 

The basic frameworks to be used for i-foviding evaluative information for 
decision-making are the CirP (Stufflebeam, 1971) and the Discrepancy Evaluatloa 
(Provus, 1972) models, with emph;»sis on tfee product evaluation aspect of the 
CIPP model and on Stage IV of the Provus model. 

TnfQTmgMo n for Project Pers onnel. The kind of description needed by X»roject 
iS'e'i:7irone which portrats the nature of the Project and its full range of 
activities. Such a representation makes it easier to Identify where functioning 
ha3 surpassed and fallan short of original expectations, to offer suggestions 
for Improvement, and to identify unanticipated phenomena. These purposes are 
difficult to accompllsti M thfe activities are evaluated without consideration 
of the context in which t«iey occi:i. The goal of this aspect of the evaluation 
is to assist Project personnel in identifying areas requiring attention. The 
primary techniques to be used for this aspect of the evaluation are those of 
responsive evaluation and portrayal (Stake, 1972,1974). 

Collection of Evaluative Information 

The evaluation is organized around the objectives of the Project as they were 
specified in the contract. These objectives were listed and then restated In 
terms of evaluation questions. For each question, data sources and collection 
procedures were identified and analyses outlined. 

Stude nt Ouestlonnaire . The attainment of a number of objectives can be 
measured (at least, in part) by finding out how the students for whom the 
lessons were designed used and reacted to the lessons. Therefore farly 
Hay. a quentionnaire soliciting opinions over many aspects of the PLATO les- 
sons and their usefulness was sent to the 98 students in the SBMS-UC program. 
Based on the responses of 81 students, the preliminary results show that the 
students felt that they had used the PLATO lessons about as often as other 
types of media such as films, slide-tapes, videotapes, etc., or an average of 
twice a month. (Actual usage statistics remain to be analyzed.) 

The students were als6 given the opportunity to rate each of the 29 subdlsci- 
plinc lesson groupings in the catalog on a scale of "very helpful to not at 
Si helpful" (with an option for "never used"). In every grouping except one. 
there were more students responding on the positive half of the scale than on 
the negative half. The lessons in four biochemistry subdisclplines (proteins, 
carbohydrates, acid-base, and glycolysis and TCA cycle) were rated very high. 
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with 36 to 39 respondents judging them to be helpful, and only two to four 
respondents judging them not to be helpful. Other groupings which were rat^d 
strong as being helpful were the biochemistry lessons on kinetics and cof actors, 
the pharmacology lessons on the autonomic and central nervous systems, and the 
lessons in genetics. 

The students were also asked to react to a number of statements about the 
PLATO lessons on a scale froni "strongly agree" to ^'strongly disagree". Sxmtr 
marizing briefly the areas where pronounced opinions were expressed, it was 
noted that the students definitely felt that PLATO was a useful learning tool 
and that the lessons were good supplements to the SBMS-UC curriculum document. 
They also agreed that the lessons supported a self-paced learning philosophy. 
VhlXe students were somev7hat divided over whether the lessons \jerf> too elemer^- 
tary, there was definite feeling that the lessons were not too difficult. The 
students felt that they would have used the PIJVTO system more had a larger 
number of concepts been covered. The students disagreed with a statement that 
they had used the lessons primarily to review for int<^.rnal examinations, which 
sugge&ts that the lessons were utilized for basic learning. Receiving imme-. 
dlate feedback to answerers in PLATO lessons was definitely perceived by the 
students as being useful > 

Additional Questions and Other Data Sources . A number of other sources of 
information was suggested during the process of extracting evaluation questions 
from the contract objectives. Priorities were assigned to each data-gathering 
procedure based on the importance of the question being asked and how faithful 
the data sources were likely to be. Some of the higher priority questions and 
data sources follow. 

1. To what extent do the developed lessons correspond to units within the 
SBMS-UC curriculimi? For what proportion of the curriculum units t^ould 
PLATO IV be an appropriate medium? How unified and comprehensive is the 
total set of lessons? 

An index relating developed lessons to the imits of the SBMS-UC curricultim 
will provide the evaluators a basis for making a rough estimate of the 
portion of the curriculum being addressed by PLATO lessons. Additionally, 
Project authors (primarily responsible for the PLATO lessons) and SBHS-UC 
teaching faculty (primarily responsible for the curriculum and associated 
instructional activities) were asked to estimate the fraction of the con- . 
cepts in their discipline's curriculum which were covered by PLATO lessons. 
Both groups were also asked to estimate the fraction of their discipline's 
content for which PLATO IV was an. inappropriate medium. 

2. How have defined student needs, conteiit, specialist resources, and func- 
tional capabilities of PLATO IV guided lesson development? How do the 
courseware materials respond to identified deficiencies of educationally 
disadvantaged students? 

Proposals for the development of each lesson contain justification for the 
content and methodology to be used in the lesson. These propossils will 
provide a data base for the evaluators to determine the primary goals and 
concerns of the developers as they began each lesson. The Project author 
interviews queried the area of procedures and criteria for lesson develop- . 
ment and review in order to shed additional light on these questions. 

4 
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Finally, the use of the PLATO lessons made by those students which are 
identified as edacationally disadvantaged (admitted in the U. of I. 
College of Medicine's Minority Opportunity Program) will be studied 
separately and compared with the usage for the entire basic sciences 



class. 



3 What evidence is there that courseware materials have been or could be 
" uped in clinical training, research, and graduate training programs in 
health professions education? 

Directors of various health professions education programs to Illinois 
are being interviewed to obtain their opinions of the lessons and wl»ether 
they might be useful in their programs. Those who have been responsible 
for remote site installations on the medical PLATO network are being 

asked to relate the types of people making use of the lessons and their 
purposes for using them. 
A. What has been done to encourage continuous updating of the PLATO lessons? 

The lescion development and review procedures, as described in doctmenta- 
tions and by Project authors and directors to the course of Ifjerviews. 
will be exanitoed by the evaluators to itemize these aspects which encour- 
age conttouous updattog. 

Effectiveness of Courseware Mater ials 

An origtoal totention described to the contract was ^'^^^^"^^^J^f ^^J^lSi^'i^ 
be assessed by compartog the achievement of basic medical sciences students to 
the IL?0-supJorted program at SBMS-UC with that of students to the mo^ tradi- 
Sonaiprogram at the Medical Center of the University of Illinois (SBMS-MC) . 
However the College of Medictoe made the PLATO IV system (including the 
jSjecJldeieloped courseware materials) available to students -rolled to . 
SBMS-MO. Obviously, this dectoion created a situation to which it is not 
possible to assess effectiveness by means of program-vs-i>rogram comparisons. 

In order to assess effectiveness of the materials, usage of lessons will be 
considered as an todependent vartobje. Usage will be operationalized in^two 
vajs: (1) total hours a student has spent to instructional materials and 
Tz) the nimber of sessions (times) on the computer. At any given PoUit^to 

ir tHii he Tjossible to derive these indices for every student. These 
SS'can ^'ib't:iJrd1oi any lesson or group of lessons. The m^^^^"-^' 
egies have been developed to analyze usage data and student performance, 
utiliztog partial correlation and analysis of covariances 

I. SBMS-UC Usage— Performance on SBMS^UC Examtoations (Level Ills) 

Each lesson has been indexed to the applicable basic science unit(s) of 
Se SbS-Sc curriculum. Each unit of the curriculum is ^J^^Sned in one 
or morr"clinical problems," which, are the major instruct tonal/ learning 
ZmZls of tte SBMS-UC program. FoUowtog each clinical problem, each 
stSeS^taLs an objective examination (Level III) over the units assigned 
TZt JroUe^! Sibscores for each basic science discipline^are reported 
Sti tWesults of these examinations. Therefore it is possible to asso- 
ciate each lesson with o^je or more Level III subscores. As each student 
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■» 100) finishes each of the clinical problems, his/her usage data for 
the applicable lessons will be totaled. 

Partial correlations will be calculated, correlating each subscore with 
the total hours of use of the applicable lessons. The "discriminant 
function" (DF) of each student (a function of MCAT scores and GPA, which 
Is used in the College admission process) will be partialled out in each 
correlation to determine the relationship of use to Level III examination 
performance.. 

If total hours or total number of sessions separates students naturally • 
into two or more groups (e.g., non-users, light users, heavy users) a one- 
way analysis of covariance will also be conducted* The Level III subscore 
will be the dependent variable, "usage group" the independent variable, 
and DF the covariate. 

II* SBMS-MC and SBMS-UC Usage — Performance on College Freshman Comprehensive 

The Project lessons have been indexed to sections of the College Freshman 
Comprehensive examination. At the end of the year, the usage data for 
each group of lessons associated with a particular Comprehensive subscore 
will be totaled for each student (N = 350, approx. 250 at SBMS-MC and 100 
at SBMS-UC). 

In a design similar to that of Section I (and with the same effectiveness 
criteria), partial correlations will be calculated, correlating each topic 
subscore wit'h the total hours of use of the applicable lessons. The DF 
(see description in Section I) and an educational program variable (SBMS-MC 
or SBMS-UC) will be partialled out in each correlation. 

If total hours or total number of sessions separates students naturally 
into two or more groups (as suggested in Section I), a two-way analysis of 
covariance will be conducted. The topic subscore will be the dependent 
variable. The independent variables will be the *'usage" grouping and the 
basic sciences program In which the students are enrolled. The DF will be 
used as a covariate » 

Data and Preliminary Results 
Preliminary results of the evaluation will be presented at the meeting. 
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